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[57] ABSTRACT 

A PC-based home security system for monitoring the envi- 
ronment surrounding a PC in order to detect suspicious or 
uncharacteristic events. The PC -based home security system 
first monitors the environment, listening and watching for a 
threshold event. When a threshold event is detected, the 
PC -based home security system then conducts close surveil- 
lance of the environment in order to detect and characterize 
additional events. When the accumulated detected events 
exceed some threshold value, the PC-based home security 
system determines that a suspicious or uncharacteristic set of 
events has occurred, diagnoses those events, and takes a 
remedial action appropriate to the diagnosed set of suspi- 
cious circumstances. 

61 Claims, 5 Drawing Sheets 
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PERSONAL COMPUTER-BASED HOME 
SECURITY SYSTEM 



TECHNICAL FIELD 

This invention relates generally to home security systems 
and, in particular, to a personal computer-based home secu- 
rity system. 

BACKGROUND OF THE INVENTION 

10 

Along with the rapid increase in processor speeds, 
memory size, and disk capacity in commonly available 
personal computers ("PCs"), the types and capabilities of 
standard input/output devices included in PCs have also 
begun to increase. In particular, PCs are currently routinely 15 
sold with a microphone and audio speakers along with the 
software and hardware components required to capture 
sound through the microphone and store the captured sound 
in data files on a magnetic disk. The PC user can purchase 
any number of software packages that allow the user to edit 20 
and play back the recorded sound through the audio speak- 
ers. 

Electronic home security systems have been sold in the 
consumer market for many years. These home security 
systems normally include a variety of sensors, including 25 
photo detectors, motion detectors, and sound detectors, 
along with a microprocessor and driving programs that 
coordinate monitoring of the sensors that analyze data 
collected through monitoring of sensors to detect suspicious 
or uncharacteristic events, and that can effect certain reme- 30 
dial actions in response to detected events. These home 
security systems are often expensive, and require extensive 
installation procedures, particularly of the sensing devices. 

SUMMARY OF THE INVENTION 35 

The present invention provides a personal computer- 
based home security system, implemented as a software 
program, that runs on commercially available personal com- 
puters. In one embodiment, the personal computer-based 40 
home security system monitors an environment to detect and 
remedy unusual circumstances that occur in the environ- 
ment. This personal computer-based home security system 
includes a monitoring routine that detects threshold events 
that indicates a change in the environment. When such a 45 
change has been detected, the personal computer-based 
home security system launches a close surveillance routine. 
The close surveillance routine closely monitors the environ- 
ment to detect, characterize, and record events that occur in 
the environment. When the close surveillance routine 50 
detects sufficient events to determine that a suspicious set of 
circumstances has occurred in the environment, the personal 
computer-based home security system calls a remedy rou- 
tine to diagnose the suspicious set of circumstances and 
initiate an appropriate remedial action consistent with the 55 
diagnosis. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 displays a simple schematic drawing of the internal 
components of a PC. 60 

FIG. 2 displays a flow control diagram for a PC-based 
home security system. 

FIG. 3 displays example event tables. 

FIG. 4 displays example detected event tables. 65 

FIG. 5 displays an example diagnoses table. 

FIG. 6 displays an example remedy table. 
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DETAILED DESCRIPTION OF THE 
INVENTION 

The present invention provides a PC-based home security 
system. In one embodiment, the PC-based home security 
system monitors the environment within a home to detect 
threshold events that may warrant closer examination. Fol- 
lowing detection of a threshold event, the home security 
system then conducts a close surveillance of the home 
environment to detect suspicious or uncharacteristic 
circumstances, diagnoses those circumstances where 
possible, and initiates remedial action in the case that the 
diagnosed circumstances are of a serious nature. The 
embodiment may be implemented on standard, commonly 
available PCs that already include a microphone or that 
include both a microphone and a video camera. The 
PC-based home security system of some embodiment of the 
present invention is thus an easy-to-install and inexpensive 
software program that runs on commonly available PCs. 

FIG. 1 displays a simple schematic drawing of the internal 
components of a PC. A PC includes a CPU 101, memory 
102, a hard disk nonvolatile data storage device 103, various 
input/output devices 104-105, and one or more internal 
buses 106 that enable the various components to exchange 
data. Software programs are executed by the CPU, which 
fetches and executes the instructions of the program stored 
in memory 102. Permanent copies of the software programs 
are stored on the hard disk 103 and transferred to memory 
prior to execution. Each separate hardware controller 104 
and 105 interfaces with one of a variety of different types of 
input/output devices, including keyboard, mouse, a 
microphone, a video camera, audio speaker, a printer, a fax, 
and a modem. Under the direction of executing software 
programs, the input/output device controllers enable transfer 
of data from input devices over the internal bus to memory 
and transfer of data over the internal bus from memory to 
output devices. A software program can direct, for example, 
a microphone to record the sound environment of a PC and 
can direct storage of the data representing the recorded 
sound into memory and into permanent data files stored on 
the hard disk. 

FIG. 2 displays a flow control diagram for one embodi- 
ment of the PC-based home security system. The PC-based 
home security system of FIG. 2 comprises one or more 
software programs instantiated as one or more correspond- 
ing executing processes within the PC. When the home 
security system is started, it begins to monitor, in step 201, 
input data from the environmental input devices attached to 
the PC, including a microphone or a microphone and a video 
camera. The PC-based home security system continues to 
monitor this input data in step 201 until it detects a threshold 
event or, alternatively, until it times out or is interrupted. 

A threshold event is generally a discontinuity in the input 
data stream that rises above a certain threshold value. For 
example, for data input from a microphone, a threshold 
event might be an abrupt increase in the detected amplitude 
at a particular sound frequency or an increase in the average 
sound level. For a video camera, a threshold event might be 
the detection of movement against the normal background, 
a marked decrease in contrast, or a rapid change in overall 
brightness. 

When the monitoring step either stops or is interrupted, 
the home security system determines, in step 202, whether 
the termination of monitoring represents an intentional inter- 
rupt command generated by the user or represents a time-out 
either based on the length of the monitoring period or based 
on the time of day. If such an expected or intentional 
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termination is detected in step 202, the home security system A more sophisticated approach that involves adaptation to 

program returns in step 203. If the monitoring step has not the normal background environment of the house can be 

been intentionally terminated, then the home security system employed. The home security system can monitor the envi- 

determines in step 204 whether the monitoring step has ronment at a particular location for a period of time in order 

detected a threshold event. If no threshold event has been 5 to characterize the normal environment with respect to the 

detected, control returns to the monitoring step 201. If, time of day. Using this more sophisticated monitoring 

however, a threshold event has been detected, then control approach, the home security system can detect threshold 

flows to the close surveillance step 205. events that represent changes in the expected background 

y , .„ i_ environment at a given time of day. 

In the close surveillance step, the home security system io 

closely monitors and records data input from the environ- It is preferable in the home security system to process the 

mental input devices. The home security system continues to raw data input from the environmental input devices during 

closely monitor this data either until the home security close surveillance in order to enumerate, characterize, and 

system determines that a suspicious set of circumstances that time stamp various types of events. To that end, the home 

require remedial action has occurred, or until no further 15 security system may include a database of different types of 

threshold events have been detected for a certain period of events along with corresponding data patterns that charac- 

time. In step 206, the home security system determines, m0 se events. FIG. 3 displays two example event 

following the close surveillance step 205, whether suspi- tables event ta51es mc i ude a ta ble of sound events 

cious circumstances requiring remedial action have 301 and a table of video events 302. In the description of an 

occurred. If not, control returns to the monitoring step 201. 20 eyent ifl eacfa rQW of eacfa ^ a numeric k for 

If, however, suspicious circumstances have been detected. , , , . .. r ., 

.t_ . -irtr, t_ j- the event, along with a character string representation of the 

then, in step 207, the home security system diagnoses those A . / . , „ ' ° ** . . A , 

' -c -ui j . t "It. • * event, is combined with a pattern characterizing the event 

circumstances, if possible, and takes the appropriate reme- , . r , , , . , , . r i b 

dial action a avent y formula by which the seventy of the event can 

25 be calculated from the recorded data. For example, in the 

In one embodiment of the invention, remedial action sound table 301, the first event has a key value 303 of "1", 

generally involves connecting via the voice enabled FAX/ a character string representation 304 of "glass breaking," a 

modem included in the PC to an outside telephone number data pattern stored in the file "gbFFT.dat" 305, and the 

and transferring over the connection one or more of a severity formula "A" 306 
number of stored messages, depending on the nature of the 

receiving party and on the diagnosis of the suspicious The key is a simple numeric designation of the event type, 

circumstances. For example, if a fax machine is called The data pattern stored in the file depends on the method by 

following detection of unusual sounds, then the home secu- which the home security system processes input data in 

rity system may send a fax -based message. If, on the other order to recognize patterns. In the case of sound data, for 

hand, the home security system elects to call a police station ^ example, recorded data can be processed via a Fast Fourier 

in response to diagnosing the presence of an intruder, then Transform to provide the amplitudes at various discrete 

the home security system may broadcast through the modem characteristic frequencies as a function of time. Thus, a 

a voice message stored as a voice data file on the hard disk recorded event can be processed using a Fast Fourier Trans- 

of the PC that informs the police of the address of the house form to pro d U ce the pattern of amplitudes at characteristic 

and a warning that an intruder is present. The activities 4o frequencies for sample times within a time period, and that 

conducted by the home security system in steps 201, 205, resultin n cao be ared to stored tterDS ^ the 

and 207 will be discussed in greater detail below. S0Ufld 

in order to choose an event type that most 

In the monitoring step 201, the home security system closely corresponds to the recorded sound input, 

essentially listens and watches through the microphone and ™ , . . t 

video camera for significant change! to the normal back- 45 f ^ b p 0me f"™* S ? stem ma J ff° St0 / e a 

ground environment within the home. Many different crite- ^rmula for calculating from recorded input data a seventy 

ria may be used to detect these changes. For example, in the metnc that corresponds to a perceived seriousness of the 

case of sound data obtained through the microphone, a sharp recorded event. For example, in the case of a glass breaking 

rise in the overall sound level within the home above some event > tf 11 were possible for the microphone to detect the 

threshold sound level value might be interpreted by the 50 sound of S 1 ^ breaking in a neighbors house several 

security system as a threshold event. Similarly, in the case of hundred feet away from the house being monitored, then 

the video camera input, a rapid change from darkness to perhaps the severity formula would be a simple function of 

lightness within the home or the detection of a large object the overall amplitude or volume recorded during the glass 

moving within the field of the video camera over a certain breaking event, so that only events loud enough to have 

period of time may be considered by the home security 55 occurred within the house being monitored are designated as 

system to be a threshold event. These threshold events are being serious. The example sound table 301 includes addi- 

not immediately perceived by the home security system to tional example events, including a footstep 307, the sound of 

be suspicious or uncharacteristic. They simply trigger a door being kicked open 308, and the sound of a light 

increased surveillance by the home security system for a switch being clicked or turned on or off 309. In similar 

certain period of time in order to detect and record a number 60 fashion, the video event table 302 includes a movement 

of events. In step 201, the input data may be temporarily event 310, a contrast change event 311, and a dark- to-light 

recorded in a circular buffer so that the home security system event 312. Such tables would include a miscellaneous or 

can append the recorded data just prior to the threshold event catch-all type of event to represent events that cannot be 

to data recorded subsequently in the close surveillance step characterized as belonging to one of the narrow, predeter- 

205 in order to have an entire record of the time period just 65 mined events such as glass breaking or footsteps. A default 

before the threshold event up until close surveillance is severity formula may be assigned to these unrecognized 

discontinued. events that may be subsequendy changed by a user of the 
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security system. Several different types of unrecognized 
events may be included in event tables, and unrecognized 
events may be associated with generalized data patterns that 
would serve to distinguish one unrecognized event type 
from the other unrecognized event types. 

In the close surveillance step 205, the home security 
system closely monitors and records the input data to detect 
events and to characterize and store the detected events into 
detected event tables. FIG. 4 displays example detected 
event tables. There is a sound event table 401 and a video 
event table 402, In both tables, events are classified accord- 
ing to a key for the event. The keys are defined in the sound 
and video event tables of FIG. 3. The classifications of 
events occur in columns 403 and columns 404 of the sound 
and video detected event tables. Along with each event 
detected by the home security system, the time that the event 
started and the time that the event ended, t start and i endi are 
stored along with the calculated severity of the event in 
columns 404, 405, and 406 of the sound detected event table 
and in columns 407, 408, and 409 of the video detected 
event table. The recorded data may be stored in ".AVI" and 
".WAV" files or in specially formatted files on either the hard 
disk of the PC or on secondary non-volatile storage devices 
like floppy drives or zip drives. 

In the close surveillance step 205, the home security 
system may employ more than one executing process. A 
single process can, for example, closely monitor the input 
data and detect the starting point for events by detecting 
abrupt discontinuities in the input data. This process can 
then store records in the detected event tables that include 
only the starting time for the event. A second process can 
then process the detected event tables by looking up the 
starting times stored by the first process, using the stored 
patterns in the sound event and video event tables shown in 
FIG. 3 to characterize or pattern match the recorded events 
with known events and to calculate a severity for each 
recorded event. In the example shown in FIG. 4, the sur- 
veillance system has detected and characterized seven dif- 
ferent types of events. At a starting time of 0, the home 
security system detected the sound of breaking glass and 
stored an entry 410 in the detected event table to correspond 
to that event. The home security system next detected the 
sound of four footsteps, stored in the detected event table in 
entries 411-414. Next, the home security system simulta- 
neously detected the sound of a light switch being clicked on 
415, as well as a dark-to-light event 416 detected from input 
video data and stored in the video detected event table. 

Thus, the close surveillance step 205 both records the 
input data as well as processes the data in order to charac- 
terize discrete events that occur during close surveillance. 
The close surveillance step may continue for some set period 
of time or until either sufficient evidence has been collected 
to characterize the accumulated events as being suspicious 
and requiring remedial action or until no further events have 
been detected for a prolonged period of time. 

The close surveillance step 205, like the monitoring step 
201, will generally make a threshold determination based on 
the events detected and stored in the detected events tables 
shown in FIG. 4. The surveillance step can determine the 
threshold of suspicion by first computing a computed events 
metric and then comparing that computed events metric to a 
threshold value for the metric. When the computed events 
metric exceeds the threshold value, then the close surveil- 
lance step would indicate that a suspicious set of circum- 
stances has occurred. 



The following three equations show three different types 
of computed events metrics that can be employed by the 
close surveillance step: 

mmN a +N v (1) 
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where: 

m=computed events metric; 

N^-number of detected sound events; 

N v -number of detected video events; 

S ; =severity of detected sound event i; 

S^-severity of detected sound event j; 

a^correlation between type of sound event i and type of 

sound event j; 
AT JMfV =length of the surveillance period; 
AT tf =expected time lapse between sound event and video 

event; and 

ATy-actual time lapse between sound event and, video 
event. 

The first computed events metric is simply the sum of the 
number of sound events and video events detected. Thus, 
using this simple metric, if more sound and video events 
have been detected than some threshold value, the close 
surveillance system indicates that a suspicious set of cir- 
cumstances has occurred. The second computed events 
metric formula is the combined sum of the sum of the 
severities of the events detected for the sound input device 
and the sum of the severities of the detected video events. 
Thus, if this second computed events metric is used, the 
close surveillance step will perceive suspicious circum- 
stances to have occurred when the accumulated severities of 
detected events exceeds some threshold value. Finally, a 
more sophisticated computed events metric, shown in equa- 
tion (3), might take into account the accumulated severities 
for the detected events along with an additional term that 
correlates the different sound events with the different video 
events that have been detected. For this formula, a table of 
event-type correlations would be maintained by the home 
security system along with expected time lapses between 
pairs of events. For example, the expected time lapse 
between the click of a light switch and a dark-to -light video 
event would be essentially 0. On the other hand, the 
expected time lapse between the sound of breaking glass and 
the detection of movement might be something on the order 
of 2 or 3 minutes, if not longer. Even more sophisticated 
computed events metrics can be employed. 

If the close surveillance step determines that the com- 
puted events metric exceeds a certain threshold, and there- 
fore perceives that a set of suspicious circumstances has 
occurred in the house, then the remedy step 207 is called by 
the home security system. The step may employ diagnoses 
and remedies of various sophistications and complexities. In 
the preferred embodiment, the remedy step attempts to 
correlate events detected by the close surveillance step to 
determine a general diagnosis of the suspicious 
circumstances, and then makes one or more telephone calls 
depending on the resulting diagnosis. 

FIG. 5 displays one embodiment of a stored diagnosis 
table that is used by the remedy step to diagnose the 
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sequence of events that have occurred. There are a variety of 
different forms and underlying algorithms that can be 
employed for this diagnosis. In an example shown in FIG. 5, 
the diagnosis table 501 includes a diagnosis key 502 and an 
event sequence symbolic description 503 for each possible 5 
diagnosis. For example, the first entry indicates that a 
diagnosis with key 1 corresponds to detection of either an 
event of type 1 or type 3, where the event types are defined 
in the key column of the event tables of FIG. 3, followed by 
multiple events of type 2, followed by an event of type 103, 10 
followed by an event of type 4. With reference to the event 
tables of FIG. 3, this first diagnosis represents detection of 
breaking glass or the sound of a door being kicked in, 
followed by a number of footsteps, followed by detection of 
a dark-to-light event by the video camera along with detec- 15 
tion of the sound of a light switch being clicked. This same 
type of diagnosis, type 1, may also result, as shown in entry 
505 in table 501, from the detection of either the sound of 
breaking glass or the sound of a door being kicked in, 
followed by the detection of movement by the video camera. 20 
The diagnosis table also generally includes a miscellaneous 
or catch-all diagnosis type that describes circumstances that 
do not fit the more specific or narrowly defined diagnoses 
stored in the diagnosis table. 

The remedy step thus compares the events logged in the 25 
detected event tables of FIG. 4 with the event sequences of 
various diagnoses listed in column 503 of diagnosis table 
501 in order to match the accumulated events detected by the 
close surveillance step with one or more diagnoses for what 
has happened within the home. The remedy step then 30 
employs a remedy table to determine what action to take in 
response to the diagnosed circumstances. 

FIG. 6 displays an example remedy table. The remedy 
table 601 includes columns for the character string repre- 
sentation of a diagnosis 602, for the key or type of the 35 
diagnosis 603 corresponding to one of the diagnosis keys 
stored in column 502 in table 501, for a telephone number 
604, for a type of receiver 605, and for a message 606. 
Continuing with the example used above, the first row or 
entry of the remedy table 607 indicates that the diagnosis 40 
having a key value of 1 is described as "intruder/' that the 
telephone number 392-4566 should be called by the home 
security system when an "intruder" diagnosis has been made 
by the home security system, that a voice -type message 
should be transmitted to this telephone number, and that the 45 
voice-type message is contained in the file "intrdr.wav." 
Thus, entry 607 in the diagnosis table indicates to the 
remedy step that if the sequence of events corresponding to 
a diagnosis type of "1" has been found in the detected event 
tables of FIG. 4, then the most likely diagnosis is that an 50 
intruder has entered the house, that the telephone number 
corresponding to the police station should be called, and that 
a previously recorded voice message that includes the 
address of the house and an indication that it is believed that 
an intruder has broken into the house will be played once 55 
either a human or an answering machine has answered the 
telephone. As shown in the remedy table of FIG. 6, a 
particular diagnosis may have more than one entry. For 
example, entry 608 specifies a different telephone number to 
be called in the case that an intruder has broken into the 60 
house and that a fax should be sent to the fax machine that 
answers the telephone at that number. Entries 609-612 
contain the telephone numbers and messages to be trans- 
mitted to those telephone numbers in the event of detection 
by the home security system of different types of diagnosed 65 
circumstances, including a fire 609, vandalism 610 and 611, 
and the outbreak of a teenage party 612. 
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Although the present invention has been described in 
terms of the several embodiments, it is not intended that the 
invention be limited to these embodiments. Modification 
within the spirit of the invention will be apparent to those 
skilled in the art. For example, a wide variety of different 
computed events metrics might be used by the close sur- 
veillance step in order to make a threshold determination of 
suspiciousness. Such metrics might correlate events with the 
time of day that the events are detected. Different environ- 
mental input devices besides microphones and video cam- 
eras might be employed. Less expensive home security 
systems can be implemented on PCs having only a 
microphone, monitoring the environment entirely by means 
of audio data. Different types of databases with different data 
organizations can be used to store event characterizations, 
detected events, diagnoses, and remedial actions. Remedial 
actions other than phone calls can be undertaken, like, for 
instance, playing through the audio speakers a voice mes- 
sage to frighten intruders. The scope of the present invention 
is defined by the claims which follow. 

We claim: 

1. The personal computer-based home security system 
implemented as a software program that runs on commer- 
cially available personal computers that include a modem, a 
microphone, and a video camera, the personal computer- 
based home security system monitoring an environment to 
detect and remedy unusual circumstances that occur in the 
environment, the personal computer-based home security 
system comprising; 

a monitoring component configured to sample data col- 
lected by the microphone and video camera to detect, 
without human assistance, threshold events that repre- 
sent a change in the environment based on the sampled 
data; 

a close surveillance component that is launched by the 
personal computer-based home security system follow- 
ing the detection of a threshold event by the monitoring 
component, the close surveillance component more 
frequently sampling data collected by the microphone 
and video camera in order to detect, characterize, and 
record events that represent differences between the 
sampled data and data normally collected from the 
environment; and 

a remedy component that is launched by the personal 
computer-based home security when the close surveil- 
lance component has detected sufficient events to ini- 
tiate an appropriate remedial action. 

2. The personal computer-based home security system of 
claim 1 wherein the monitoring component compares audio 
data input from the microphone and video data input from 
the video camera to expected background audio and video 
data for the environment and detects as a threshold event a 
discrepancy between the input data and expected back- 
ground data greater than a threshold value. 

3. The personal computer-based home security system of 
claim 2 wherein an increase in the amplitude of input audio 
data above the amplitude of the expected background audio 
data over a short time interval is a discrepancy between the 
input data and expected background data. 

4. The personal computer-based home security system of 
claim 2 wherein detection of movement within in the input 
video data is a discrepancy between the input data and 
expected background data. 

5. The personal computer-based home security system of 
claim 2 wherein an increase in the brightness of input video 
data above the brightness of the expected background video 
data over a short time interval is a discrepancy between the 
input data and expected background data. 
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6. The personal computer-based home security system of 19. The personal computer-based home security system of 
claim 2 wherein a decrease in the brightness of input video claim 18 wherein event sequences are lists of events ordered 
data above the brightness of the expected background video by time of occurrence. 

data over a short time interval is a discrepancy between the 20. The personal computer-based home security system of 

input data and expected background data. 5 claim 18, further including a remedy collection that contains 

7. The personal computer-based home security system of indications of remedial actions that should be initiated 
claim 2 wherein a decrease in the contrast of input video data following determination of a particular diagnosis of events 
above the contrast of the expected background video data that have occurred in the environment and that have been 
over a short time interval is a discrepancy between the input detected by the close surveillance component. 

data and expected background data. 10 21. The personal computer-based home security system of 

8. The personal computer-based home security system of claim 20 wherein the remedy component compares the 
claim 2 wherein the input audio data and input video data are events that have been detected by the close surveillance 
correlated with the time of day of input and compared to component to the different event and event sequences stored 
audio data and video data expected for that time of day. in the diagnosis collection in order to match the detected 

9. The personal computer-based home security system of 15 events with a most likely diagnosis, selects actions from the 
claim 2 wherein the input audio data and input video data are remedy collection consistent with the most likely diagnosis, 
correlated with the time of day and the day of the week of and initiates the selected actions. 

input and compared to audio data and video data expected 22. The personal computer-based home security system of 

for that time of day and day of the week. claim 1 wherein a remedial action directs the personal 

10. The personal computer-based home security system of 20 computer-based home security system to call a specific 
claim 2 wherein the personal computer-based home security telephone number via the modem and send a specific mes- 
system is first trained by exposing it to the environment so sage through the modem to a receiving party, the remedy 
that the personal computer-based home security system can collection storing an indication of the type of receiving party 
detect and store a representation of the expected background to expect for each telephone number, including a human, a 
input for the environment. 25 fax machine, and another modem. 

11. The personal computer-based home security system of 23. The personal computer-based home security system of 
claim 1, further including data patterns that define and claim 1, further including data patterns that defines and 
categorize types of events. categorize types of events. 

12. The personal computer-based home security system of 24. The personal computer-based home security system of 
claim 11 wherein the close surveillance component monitors 30 claim 23 wherein the close surveillance component monitors 
and records audio data input from the microphone and video and records audio data input from the microphone, detects 
data input from the video camera, detects differences in the differences in the input data from expected background, 
input data from expected background, compares the detected compares the detected differences with input patterns to 
differences with input patterns to determine the type of event determine the type of event that produced the differences, 
that produced the differences, and computes a metric that 35 and computes a metric that describes a suspicion level 
describes a suspicion level corresponding to the detected corresponding to the detected events. 

events. 25. The personal computer-based home security system of 

13. The personal computer-based home security system of claim 24 where the computed metric that describes a sus- 
claim 12 where the computed metric that describes a sus- picion level is a sum of the number of different events 
picion level is a sum of the number of different events 40 detected by the close surveillance component. 

detected by the close surveillance component. 26. The personal computer-based home security system of 

14. The personal computer-based home security system of claim 24 including an event collection that contains infor- 
claim 12 including an event collection that contains infor- mation for each type of event that indicates to the close 
mation for each type of event that indicates to the close surveillance component how to compute a severity metric 
surveillance component how to compute a severity metric 45 for that event, the information used by the close surveillance 
for that event, the information used by the close surveillance component to compute a severity metric in the indicated 
component to compute a severity metric in the indicated manner for each detected event. 

manner for each detected event. 27. The personal computer-based home security system of 

15. The personal computer-based home security system of claim 24 where the computed metric that describes a sus- 
claim 12 where the computed metric that describes a sus- 50 picion level is a sum of the of the severity metrics computed 
picion level is a sum of the of the severity metrics computed for the different events detected by the close surveillance 
for the different events detected by the close surveillance component. 

component. 28. The personal computer-based home security system of 

16. The personal computer-based home security system of claim 24 where the computed metric that describes a sus- 
claim 12 where the computed metric that describes a sus- 55 picion level is a sum of the of the severity metrics computed 
picion level is a sum of the of the severity metrics computed for the different events detected by the close surveillance 
for the different events detected by the close surveillance component along with a sum of correlations between pairs 
component along with a sum of correlations between pairs of events. 

of events. 29. The personal computer-based home security system of 

17. The personal computer-based home security system of 60 claim 24 wherein the close surveillance component stores 
claim 11 wherein the close surveillance component stores indications of detected events into a detected event collec- 
indications of detected events into detected event collec- tion. 

tions. 30. The personal computer-based home security system 

18. The personal computer-based home security system of implemented as a software program that runs on commer- 
claim 1, further including a diagnosis collection that con- 65 daily available personal computers that include a modem, a 
tains indications of different types of diagnoses correlated microphone, the personal computer-based home security 
with different event and event sequences. system monitoring an environment to detect and remedy 
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unusual circumstances that occur in the environment, the 
personal computer-based home security comprising: 

a monitoring component configured to sample data col- 
lected by the microphone to detect, without human 
assistance, threshold events that represent a change in 5 
the environment based on the sampled data; 

a close surveillance component that is launched by the 
personal computer-based home security system follow- 
ing the detection of a threshold event by the monitoring 
component, the close surveillance component more 10 
frequently sampling data collected by the microphone 
in order to detect, characterize, and record events that 
represent differences between the sampled data and 
data normally collected from the environment; and 

a remedy component that is launched by the personal 
computer-based home security when the close surveil- 
lance component has detected sufficient events to ini- 
tiate an appropriate remedial action. 

31. The personal computer-based home security system of 
claim 30 wherein the monitoring component compares audio 
data input from the microphone to expected background 20 
audio data for the environment and detects as a threshold 
event a discrepancy between the input audio data and 
expected background audio data greater than a threshold 
value. 

32. The personal computer-based home security system of 25 
claim 30 wherein an increase in the amplitude of input audio 
data above the amplitude of the expected background audio 
data over a short time interval is a discrepancy between the 
input data and expected background data. 

33. The personal computer-based home security system of 30 
claim 30 wherein the input audio data is correlated with the 
time of day of input and compared to audio data expected for 
that time of day. 

34. The personal computer-based home security system of 
claim 30 wherein the input audio data is correlated with the 35 
time of day and the day of the week of input and compared 

to audio data expected for that time of day and day of the 
week. 

35. The personal computer-based home security system of 
claim 30 wherein the personal computer-based home secu- 40 
rity system is first trained by exposing it to the environment 

so that the personal computer-based home security system 
can detect and store a representation of the expected back- 
ground input for the environment. 

36. The personal computer-based home security system of 45 
claim 30, further including a diagnosis collection that con- 
tains indications of different types of diagnoses correlated 
with different event and event sequences. 

37. The personal computer-based home security system of 
claim 36 wherein event sequences are lists of events ordered 50 
by time of occurrence. 

38. The personal computer-based home security system of 
claim 36, further including a remedy collection that contains 
indications of remedial actions that should be initiated 
following determination of a particular diagnosis of events 55 
that have occurred in the environment and that have been 
detected by the close surveillance component. 

39. The personal computer-based home security system of 
claim 38 wherein the remedy component compares the 
events that have been detected by the close surveillance 60 
component to the different event and event sequences stored 

in the diagnosis collection in order to match the detected 
events with a most likely diagnosis, selects actions from the 
remedy collection consistent with the most likely diagnosis, 
and initiates the selected actions. 65 

40. The personal computer-based home security system of 
claim 30 wherein a remedial action directs the personal 
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computer-based home security system to call a specific 
telephone number via the modem and send a specific mes- 
sage through the modem to a receiving party, the remedy 
collection storing an indication of the type of receiving party 
to expect for each telephone number, including a human, a 
fax machine, and another modem. 

41. A personal computer-based home security imple- 
mented as a software program that runs on commercially 
available personal computers that include a modem, and 
video camera, the personal computer-based home security 
system monitoring an environment to detect and remedy 
unusual circumstances that occur in the environment, the 
personal computer-based home security system comprising: 

a monitoring component configured to sample data col- 
lected by the microphone to detect, without human 
assistance, threshold events that represent a change in 
the environment based on the sampled data; 

a close surveillance component that is launched by the 
personal computer-based home security system follow- 
ing the detection of a threshold event by the monitoring 
component, the close surveillance component more 
frequently sampling data collected by the microphone 
in order to detect, characterize, and record events that 
represent differences between the sampled data and 
data normally collected from the environment; and 

a remedy component that is launched by the personal 
computer-based home security when the close surveil- 
lance component has detected sufficient events to ini- 
tiate an appropriate remedial action. 

42. The personal computer-based home security system of 
claim 41 wherein the monitoring component compares video 
data input from the video camera to expected background 
video data for the environment and detects as a threshold 
event a discrepancy between the input video data and 
expected background video data greater than a threshold 
value. 

43. The personal computer-based home security system of 
claim 41 wherein detection of movement within in the input 
video data is a discrepancy between the input data and 
expected background data. 

44. The personal computer-based home security system of 
claim 42 wherein an increase in the brightness of input video 
data above the brightness of the expected background video 
data over a short time interval is a discrepancy between the 
input data and expected background data. 

45. The personal computer-based home security system of 
claim 42 wherein a decrease in the brightness of input video 
data above the brightness of the expected background video 
data over a short time interval is a discrepancy between the 
input data and expected background data. 

46. The personal computer-based home security system of 
claim 42 wherein a decrease in the contrast of input video 
data above the contrast of the expected background video 
data over a short time interval is a discrepancy between the 
input data and expected background data. 

47. The personal computer-based home security system of 
claim 42 wherein the input video data is correlated with the 
time of day of input and compared to video data expected for 
that time of day. 

48. The personal computer-based home security system of 
claim 42 wherein the input video data is correlated with the 
time of day and the day of the week of input and compared 
to video data expected for that time of day and day of the 
week. 

49. The personal computer-based home security system of 
claim 42 wherein the personal computer-based home secu- 
rity system is first trained by exposing it to the environment 
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so that the personal computer-based home security system 56. The personal computer-based home security system of 

can detect and store a representation of the expected back- claim 50 wherein the close surveillance component stores 

ground input for the environment indications of detected events into detected event collec- 

50. The personal computer-based home security system of lions. 

claim 41, further including data patterns that define and 5 57. The personal computer-based home security system of 

categorize types of events. claim 56j fu rther including a diagnosis collection that con- 

51. The personal computer-based home security system of tains indications of different types of diagnoses correlated 
claim 50 wherein the close surveillance component monitors ^ differect event and event sequences. 

and records video data input from the video camera, detects 58 ^ al ter . based home securil tem of 

differences m the input data from expected background, 10 , . u 4 v 4 c \ , , 

» < * j • cc .A • . J? . claim 57 wherein event sequences are lists of events ordered 

compares the detected differences with mput patterns to , A . c n 

j * * * c * * j j fv. j-ir by time of occurrence, 

determine the type of event that produced the differences, -« ™, , , 

and computes a metric that describes a suspicion level 59. The personal computer-based home security system of 

corresponding to the detected events. claim 57 ' further indudlc S a remedy collection that contains 

52. The personal computer-based home security system of 15 ^^ications of remedial actions that should be initiated 
claim 51 where the computed metric that describes a sus- following determination of a particular diagnosis of events 
picion level is a sum of the number of different events mat nave occurred in the environment and that have been 
detected by the close surveillance component. detected by the close surveillance component. 

53. The personal computer-based home security system of *0. The personal computer-based home security system of 
claim 51 including an event collection that contains infor- 20 claim 59 wherein the remedy component compares the 
mation for each type of event that indicates to the close events that have been detected by the close surveillance 
surveillance component how to compute a severity metric component to the different event and event sequences stored 
for that event, the information used by the close surveillance in the diagnosis collection in order to match the detected 
component to compute a severity metric in the indicated events with a most likely diagnosis, selects actions from the 
manner for each detected event, 25 remedy collection consistent with the most likely diagnosis, 

54. The personal computer-based home security system of and initiates the selected actions. 

claim 51 where the computed metric that describes a sus- 61. The personal computer-based home security system of 

picion level is a sum of the of the severity metrics computed claim 41 wherein a remedial action directs the personal 

for the different events detected by the close surveillance computer-based home security system to call a specific 

component. 30 telephone number via the modem and send a specific mes- 

55. The personal computer-based home security system of sage through the modem to a receiving party, the remedy 
claim 51 where the computed metric that describes a sus- collection storing an indication of the type of receiving party 
picion level is a sum of the of the severity metrics computed to expect for each telephone number, including a human, a 
for the different events detected by the close surveillance fax machine, and another modem. 

component along with a sum of correlations between pairs 35 

of events. ***** 
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"microphone" 
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and a video camera - 
~ video camera - 
» video camera - 
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